Feature: Book Genie Mode - Automated Testing with Detailed Reporting
File name: BookGenie.feature
Relative path: features/BookGenie.feature
Scenario Outline: Book Genie Response Validation with Citation Verification
When
I open the Creative Workspace login page
00:00:03.440
Then
I should see the homepage
00:00:00.042
When
I click on the mode selection dropdown
00:00:00.000
Then
I can see the "BookGenieQA" mode
00:00:00.000
And
I select the "BookGenieQA" mode
00:00:00.000
Then
I type "Suggest 5 books on christmas" on chat input element
00:00:04.084
And
I wait for AI to complete thinking
00:00:06.770
And
I validate the response is visible for "Suggest 5 books on christmas"
00:00:00.007
And
I extract book data from BookGenie response
00:00:00.006
And
I save BookGenie book data to Excel file for query "Suggest 5 books on christmas"
00:00:00.000
+ Show Error
Error: No book data extracted from BookGenie response.
at CustomWorld.(anonymous) (/home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/steps/bookGenie.step.ts:46:15)
at /home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/steps/bookGenie.step.ts:8:71
at __awaiter (/home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/steps/bookGenie.step.ts:4:12)
at CustomWorld.(anonymous) (/home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/steps/bookGenie.step.ts:59:12)
And
I validate each book individually against Excel file for query "Suggest 5 books on christmas"
0s
And
I validate that reason texts match citation texts with 80% similarity
0s
And
I generate detailed citation validation report
0s
Then
I verify database connectivity
0s
And
database should contain 2020 books
0s
And
I validate extracted books individually against database
0s
And
I validate BookGenie response relevance with AI for query "Suggest 5 books on christmas"
0s
=== SCENARIO LOGS: Book Genie Response Validation with Citation Verification === [6:17:11 PM] Navigating to Creative Workspace login page [6:17:15 PM] Homepage visibility: false [6:17:15 PM] ℹ️ Typing query: "Suggest 5 books on christmas" [6:17:15 PM] ℹ️ Preparing to type query: "Suggest 5 books on christmas" [6:17:15 PM] ℹ️ Waiting for chat input to be visible [6:17:17 PM] ℹ️ Clearing chat input [6:17:17 PM] ℹ️ Typing query: "Suggest 5 books on christmas" [6:17:18 PM] ℹ️ Pressing Enter to submit query [6:17:19 PM] ✅ Query submitted successfully: "Suggest 5 books on christmas" [6:17:19 PM] ℹ️ Waiting for AI to complete thinking... [6:17:19 PM] 📋 WAITING FOR AI RESPONSE [6:17:19 PM] ℹ️ Looking for AI thinking indicator... [6:17:19 PM] ℹ️ Waiting for thinking indicator to appear (max 2 minutes) [6:17:21 PM] ✅ ✓ AI thinking indicator appeared - AI is processing the request [6:17:21 PM] ℹ️ Waiting for thinking indicator to disappear (max 5 minutes) [6:17:23 PM] ✅ ✓ AI thinking completed - Response should be ready [6:17:23 PM] ℹ️ Allowing additional time for response rendering [6:17:26 PM] ✅ AI response wait process completed [6:17:26 PM] ℹ️ Validating response for query: "Suggest 5 books on christmas" [6:17:26 PM] 📋 VALIDATING RESPONSE FOR QUERY: "Suggest 5 books on christmas" [6:17:26 PM] ℹ️ Locating the latest response in chat [6:17:26 PM] ℹ️ Waiting for response to be visible (max 30 seconds) [6:17:26 PM] ✅ Response received - Length: 608 characters [6:17:26 PM] ℹ️ Response preview: Watch Me WorkInterpreting context for your query...That sounds like a lovely theme! To help you find... [6:17:26 PM] ℹ️ Extracting book data from response... [6:17:26 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE [6:17:26 PM] ℹ️ Locating BookGenie response container [6:17:26 PM] ℹ️ Extracting HTML content from response [6:17:26 PM] ℹ️ HTML content extracted - Length: 7560 characters [6:17:26 PM] ℹ️ Parsing HTML to extract book data [6:17:26 PM] ✅ SUCCESSFULLY EXTRACTED 0 BOOKS [6:17:26 PM] 📋 EXTRACTED BOOKS SUMMARY [6:17:26 PM] 📊 Total books extracted: 0 [6:17:26 PM] 📊 Average relevance score: 0% [6:17:26 PM] ✅ Extracted 0 books from response === END LOGS ===
Scenario Logs: Book Genie Response Validation with Citation Verification
[6:17:11 PM] Navigating to Creative Workspace login page
[6:17:15 PM] Homepage visibility: false
[6:17:15 PM] ℹ️ Typing query: "Suggest 5 books on christmas"
[6:17:15 PM] ℹ️ Preparing to type query: "Suggest 5 books on christmas"
[6:17:15 PM] ℹ️ Waiting for chat input to be visible
[6:17:17 PM] ℹ️ Clearing chat input
[6:17:17 PM] ℹ️ Typing query: "Suggest 5 books on christmas"
[6:17:18 PM] ℹ️ Pressing Enter to submit query
[6:17:19 PM] ✅ Query submitted successfully: "Suggest 5 books on christmas"
[6:17:19 PM] ℹ️ Waiting for AI to complete thinking...
[6:17:19 PM] 📋 WAITING FOR AI RESPONSE
[6:17:19 PM] ℹ️ Looking for AI thinking indicator...
[6:17:19 PM] ℹ️ Waiting for thinking indicator to appear (max 2 minutes)
[6:17:21 PM] ✅ ✓ AI thinking indicator appeared - AI is processing the request
[6:17:21 PM] ℹ️ Waiting for thinking indicator to disappear (max 5 minutes)
[6:17:23 PM] ✅ ✓ AI thinking completed - Response should be ready
[6:17:23 PM] ℹ️ Allowing additional time for response rendering
[6:17:26 PM] ✅ AI response wait process completed
[6:17:26 PM] ℹ️ Validating response for query: "Suggest 5 books on christmas"
[6:17:26 PM] 📋 VALIDATING RESPONSE FOR QUERY: "Suggest 5 books on christmas"
[6:17:26 PM] ℹ️ Locating the latest response in chat
[6:17:26 PM] ℹ️ Waiting for response to be visible (max 30 seconds)
[6:17:26 PM] ✅ Response received - Length: 608 characters
[6:17:26 PM] ℹ️ Response preview: Watch Me WorkInterpreting context for your query...That sounds like a lovely theme! To help you find...
[6:17:26 PM] ℹ️ Extracting book data from response...
[6:17:26 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:17:26 PM] ℹ️ Locating BookGenie response container
[6:17:26 PM] ℹ️ Extracting HTML content from response
[6:17:26 PM] ℹ️ HTML content extracted - Length: 7560 characters
[6:17:26 PM] ℹ️ Parsing HTML to extract book data
[6:17:26 PM] ✅ SUCCESSFULLY EXTRACTED 0 BOOKS
[6:17:26 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:17:26 PM] 📊 Total books extracted: 0
[6:17:26 PM] 📊 Average relevance score: 0%
[6:17:26 PM] ✅ Extracted 0 books from response
Scenario Outline: Book Genie Response Validation with Citation Verification
When
I open the Creative Workspace login page
00:00:02.959
Then
I should see the homepage
00:00:00.031
When
I click on the mode selection dropdown
00:00:00.000
Then
I can see the "BookGenieQA" mode
00:00:00.000
And
I select the "BookGenieQA" mode
00:00:00.000
Then
I type "Suggest 3 books for childrens" on chat input element
00:00:03.213
And
I wait for AI to complete thinking
00:00:14.017
And
I validate the response is visible for "Suggest 3 books for childrens"
00:00:00.006
And
I extract book data from BookGenie response
00:00:00.007
And
I save BookGenie book data to Excel file for query "Suggest 3 books for childrens"
00:00:00.010
And
I validate each book individually against Excel file for query "Suggest 3 books for childrens"
00:00:00.012
CITATION VALIDATION DETAILED REPORT ================================================================================ BOOK: Henry Huggins ------------------------------------------------------------ ✅ Reason 1: PASS (100%) ❌ Reason 2: FAIL (0%) 📝 Reason: It features an average boy whose life is turned upside down when he meets a lovable puppy with a nos... 📚 Citation: Just as Henry Huggins is complaining that nothing exciting ever happens, a friendly dog sits down be... ⚠️ Citation text not found in reason: "Just as Henry Huggins is complaining that nothing exciting ever happens, a friendly dog sits down beside him and looks pleadingly at his ice-cream cone. From that moment on, the two are inseparable. But when Ribsy's original owner appears, trying to reclaim his dog, Henry's faced with the possibility of losing his new best friend. Has Klickitat Street seen the last of rambunctious Ribsy?" ✅ Reason 3: PASS (95%) BOOK: Anne of Green Gables ------------------------------------------------------------ ✅ Reason 1: PASS (100%) ✅ Reason 2: PASS (100%) ✅ Reason 3: PASS (100%) BOOK: The Secret Garden ------------------------------------------------------------ ❌ Reason 1: FAIL (0%) 📝 Reason: The book is a classic... 📚 Citation: classic literature... ⚠️ Citation text not found in reason: "classic literature" ❌ Reason 2: FAIL (0%) 📝 Reason: It features an orphaned girl who discovers a secret garden and brings it back to life... 📚 Citation: Mary discovers a secret garden, surrounded by walls and locked with a missing key. One day, with the... ⚠️ Citation text not found in reason: "Mary discovers a secret garden, surrounded by walls and locked with a missing key. One day, with the help of two unexpected companions, she discovers a way in. Is everything in the garden dead, or can Mary bring it back to life?" ✅ Reason 3: PASS (100%) SUMMARY: 6/9 reasons passed (66.7%) OVERALL STATUS: FAIL
Citation validation failed: 6/9 passed (66.7%)
Citation Validation Summary: 6/9 passed
📊 Citation Validation Report
Generated on 10/15/2025, 6:19:25 PM
📖 Henry Huggins
✅ Reason 1: 100% match
❌ Reason 2: 0% match
⚠️ Citation text not found in reason: "Just as Henry Huggins is complaining that nothing exciting ever happens, a friendly dog sits down beside him and looks pleadingly at his ice-cream cone. From that moment on, the two are inseparable. But when Ribsy's original owner appears, trying to reclaim his dog, Henry's faced with the possibility of losing his new best friend. Has Klickitat Street seen the last of rambunctious Ribsy?"
✅ Reason 3: 95% match
📖 Anne of Green Gables
✅ Reason 1: 100% match
✅ Reason 2: 100% match
✅ Reason 3: 100% match
📖 The Secret Garden
❌ Reason 1: 0% match
⚠️ Citation text not found in reason: "classic literature"
❌ Reason 2: 0% match
⚠️ Citation text not found in reason: "Mary discovers a secret garden, surrounded by walls and locked with a missing key. One day, with the help of two unexpected companions, she discovers a way in. Is everything in the garden dead, or can Mary bring it back to life?"
✅ Reason 3: 100% match
📈 Validation Summary
Total Reasons Validated: 9
Passed: 6
Failed: 3
Pass Rate: 66.7%
Overall Status: FAIL
Then
I verify database connectivity
00:00:00.329
And
database should contain 2020 books
00:00:00.176
And
I validate extracted books individually against database
00:00:00.515
And
I validate BookGenie response relevance with AI for query "Suggest 3 books for childrens"
00:00:06.378
+ Show Info
+ Show Info
PER-BOOK AI VALIDATION REPORT - Query: "Suggest 3 books for childrens"
================================================================================
OVERALL RELEVANCE SCORE: 70%
STATUS: FAIL
BOOKS ANALYZED: 3
BOOK 1: "Henry Huggins"
------------------------------------------------------------
Overall Score: 70%
SECTION SCORES:
Author Information: 70% - Analysis unavailable - API error
Publishing Date: 70% - Analysis unavailable - API error
Why Match Explanations: 70% - Analysis unavailable - API error
Relevance Scores: 70% - Analysis unavailable - API error
DETAILED FEEDBACK:
• AI analysis temporarily unavailable due to service overload
IMPROVEMENT SUGGESTIONS:
• Retry analysis when service is available
================================================================================
BOOK 2: "Anne of Green Gables"
------------------------------------------------------------
Overall Score: 70%
SECTION SCORES:
Author Information: 70% - Analysis unavailable - API error
Publishing Date: 70% - Analysis unavailable - API error
Why Match Explanations: 70% - Analysis unavailable - API error
Relevance Scores: 70% - Analysis unavailable - API error
DETAILED FEEDBACK:
• AI analysis temporarily unavailable due to service overload
IMPROVEMENT SUGGESTIONS:
• Retry analysis when service is available
================================================================================
BOOK 3: "The Secret Garden"
------------------------------------------------------------
Overall Score: 70%
SECTION SCORES:
Author Information: 70% - Analysis unavailable - API error
Publishing Date: 70% - Analysis unavailable - API error
Why Match Explanations: 70% - Analysis unavailable - API error
Relevance Scores: 70% - Analysis unavailable - API error
DETAILED FEEDBACK:
• AI analysis temporarily unavailable due to service overload
IMPROVEMENT SUGGESTIONS:
• Retry analysis when service is available
================================================================================
SUMMARY FEEDBACK:
----------------------------------------
• AI analysis service is temporarily overloaded
• Using fallback analysis with neutral scores
• Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits.
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2
Please retry in 27.101040378s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"27s"}]
OVERALL IMPROVEMENT SUGGESTIONS:
----------------------------------------
• Retry the analysis when Gemini API is less busy
• Consider running tests during off-peak hours
• Check Google AI status page for service updates
🤖 PER-BOOK AI Response Relevance Validation
Query: "Suggest 3 books for childrens"
Overall Relevance Score: 70%
Status: FAIL
Books Analyzed: 3
Individual Book Analyses
📖 Book 1: "Henry Huggins" - 70%
Author Information: 70%
Analysis unavailable - API error
Publishing Date: 70%
Analysis unavailable - API error
Why Match Explanations: 70%
Analysis unavailable - API error
Relevance Scores: 70%
Analysis unavailable - API error
Improvement Suggestions:
💡 Retry analysis when service is available
📖 Book 2: "Anne of Green Gables" - 70%
Author Information: 70%
Analysis unavailable - API error
Publishing Date: 70%
Analysis unavailable - API error
Why Match Explanations: 70%
Analysis unavailable - API error
Relevance Scores: 70%
Analysis unavailable - API error
Improvement Suggestions:
💡 Retry analysis when service is available
📖 Book 3: "The Secret Garden" - 70%
Author Information: 70%
Analysis unavailable - API error
Publishing Date: 70%
Analysis unavailable - API error
Why Match Explanations: 70%
Analysis unavailable - API error
Relevance Scores: 70%
Analysis unavailable - API error
Improvement Suggestions:
💡 Retry analysis when service is available
Summary Feedback
• AI analysis service is temporarily overloaded
• Using fallback analysis with neutral scores
• Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits. * Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2 Please retry in 27.101040378s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"27s"}]
Overall Improvement Suggestions
💡 Retry the analysis when Gemini API is less busy
💡 Consider running tests during off-peak hours
💡 Check Google AI status page for service updates
=== SCENARIO LOGS: Book Genie Response Validation with Citation Verification ===
[6:17:27 PM] Navigating to Creative Workspace login page
[6:17:30 PM] Homepage visibility: false
[6:17:30 PM] ℹ️ Typing query: "Suggest 3 books for childrens"
[6:17:30 PM] ℹ️ Preparing to type query: "Suggest 3 books for childrens"
[6:17:30 PM] ℹ️ Waiting for chat input to be visible
[6:17:32 PM] ℹ️ Clearing chat input
[6:17:32 PM] ℹ️ Typing query: "Suggest 3 books for childrens"
[6:17:33 PM] ℹ️ Pressing Enter to submit query
[6:17:33 PM] ✅ Query submitted successfully: "Suggest 3 books for childrens"
[6:17:33 PM] ℹ️ Waiting for AI to complete thinking...
[6:17:33 PM] 📋 WAITING FOR AI RESPONSE
[6:17:33 PM] ℹ️ Looking for AI thinking indicator...
[6:17:33 PM] ℹ️ Waiting for thinking indicator to appear (max 2 minutes)
[6:17:34 PM] ✅ ✓ AI thinking indicator appeared - AI is processing the request
[6:17:34 PM] ℹ️ Waiting for thinking indicator to disappear (max 5 minutes)
[6:17:44 PM] ✅ ✓ AI thinking completed - Response should be ready
[6:17:44 PM] ℹ️ Allowing additional time for response rendering
[6:17:47 PM] ✅ AI response wait process completed
[6:17:47 PM] ℹ️ Validating response for query: "Suggest 3 books for childrens"
[6:17:47 PM] 📋 VALIDATING RESPONSE FOR QUERY: "Suggest 3 books for childrens"
[6:17:47 PM] ℹ️ Locating the latest response in chat
[6:17:47 PM] ℹ️ Waiting for response to be visible (max 30 seconds)
[6:17:47 PM] ✅ Response received - Length: 2030 characters
[6:17:47 PM] ℹ️ Response preview: Watch Me WorkInterpreting context for your query...Retrieving Relevant Books for 'books suitable for...
[6:17:47 PM] ℹ️ Extracting book data from response...
[6:17:47 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:17:47 PM] ℹ️ Locating BookGenie response container
[6:17:47 PM] ℹ️ Extracting HTML content from response
[6:17:47 PM] ℹ️ HTML content extracted - Length: 25302 characters
[6:17:47 PM] ℹ️ Parsing HTML to extract book data
[6:17:47 PM] ✅ SUCCESSFULLY EXTRACTED 3 BOOKS
[6:17:47 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:17:47 PM] 📚 1. "Henry Huggins" - Score: 100%%
[6:17:47 PM] 📚 2. "Anne of Green Gables" - Score: 100%%
[6:17:47 PM] 📚 3. "The Secret Garden" - Score: 100%%
[6:17:47 PM] 📊 Total books extracted: 3
[6:17:47 PM] 📊 Average relevance score: 100%
[6:17:47 PM] ✅ Extracted 3 books from response
[6:17:47 PM] ℹ️ Saving 3 books to Excel file: test_results/suggest_3_books_for_childrens.xlsx
[6:17:47 PM] 📋 SAVING BOOKS TO EXCEL FILE
[6:17:47 PM] ℹ️ Target file path: test_results/suggest_3_books_for_childrens.xlsx
[6:17:47 PM] ℹ️ Exporting 3 books to Excel format
[6:17:47 PM] ✅ Book data saved to: test_results/suggest_3_books_for_childrens.xlsx
[6:17:47 PM] 📊 File verified - Size: 16.87 KB
[6:17:47 PM] ✅ ✓ Excel file creation verified successfully
[6:17:47 PM] 📋 STARTING INDIVIDUAL BOOK VALIDATION (3 books) for query: "Suggest 3 books for childrens"
[6:17:47 PM] 📋 📚 VALIDATING BOOK 1: "Henry Huggins"
[6:17:47 PM] ℹ️ Starting comprehensive book validation
[6:17:47 PM] ℹ️ Validating title: "Henry Huggins" vs expected: "Henry Huggins"
[6:17:47 PM] ✅ ✅ TITLE VALIDATION PASSED: "Henry Huggins"
[6:17:47 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:17:47 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:17:47 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:17:47 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:17:47 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:17:47 PM] ✅ 🎉 BOOK 1 VALIDATION: ALL CHECKS PASSED!
[6:17:47 PM] ℹ️ Book 1 validation completed
[6:17:47 PM] 📋 📚 VALIDATING BOOK 2: "Anne of Green Gables"
[6:17:47 PM] ℹ️ Starting comprehensive book validation
[6:17:47 PM] ℹ️ Validating title: "Anne of Green Gables" vs expected: "Anne of Green Gables"
[6:17:47 PM] ✅ ✅ TITLE VALIDATION PASSED: "Anne of Green Gables"
[6:17:47 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:17:47 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:17:47 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:17:47 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:17:47 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:17:47 PM] ✅ 🎉 BOOK 2 VALIDATION: ALL CHECKS PASSED!
[6:17:47 PM] ℹ️ Book 2 validation completed
[6:17:47 PM] 📋 📚 VALIDATING BOOK 3: "The Secret Garden"
[6:17:47 PM] ℹ️ Starting comprehensive book validation
[6:17:47 PM] ℹ️ Validating title: "The Secret Garden" vs expected: "The Secret Garden"
[6:17:47 PM] ✅ ✅ TITLE VALIDATION PASSED: "The Secret Garden"
[6:17:47 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:17:47 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:17:47 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:17:47 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:17:47 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:17:47 PM] ✅ 🎉 BOOK 3 VALIDATION: ALL CHECKS PASSED!
[6:17:47 PM] ℹ️ Book 3 validation completed
[6:17:47 PM] 📊 VALIDATION SUMMARY: 3/3 books passed
[6:17:47 PM] ✅ ALL BOOKS VALIDATED SUCCESSFULLY!
[6:17:47 PM] 📋 🔍 VALIDATING REASON-CITATION MATCHES
[6:17:47 PM] ℹ️ Starting citation validation process...
[6:17:47 PM] 📋 🚀 STARTING COMPLETE CITATION VALIDATION WORKFLOW
[6:17:47 PM] ℹ️ Step 1: Extracting books from AI response
[6:17:47 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:17:47 PM] ℹ️ Locating BookGenie response container
[6:17:47 PM] ℹ️ Extracting HTML content from response
[6:17:47 PM] ℹ️ HTML content extracted - Length: 25302 characters
[6:17:47 PM] ℹ️ Parsing HTML to extract book data
[6:17:47 PM] ✅ SUCCESSFULLY EXTRACTED 3 BOOKS
[6:17:47 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:17:47 PM] 📚 1. "Henry Huggins" - Score: 100%%
[6:17:47 PM] 📚 2. "Anne of Green Gables" - Score: 100%%
[6:17:47 PM] 📚 3. "The Secret Garden" - Score: 100%%
[6:17:47 PM] 📊 Total books extracted: 3
[6:17:47 PM] 📊 Average relevance score: 100%
[6:17:47 PM] ℹ️ Step 2: Extracting citation texts from book elements
[6:17:47 PM] 📋 📚 EXTRACTING CITATION TEXTS FROM BOOK ELEMENTS
[6:17:47 PM] ℹ️ Step 1: Waiting for main book section
[6:17:47 PM] ℹ️ Found 7 total accordion sections
[6:17:47 PM] ℹ️ Found individual book: "1. Henry Huggins"
[6:17:47 PM] ℹ️ Found individual book: "2. Anne of Green Gables"
[6:17:48 PM] ℹ️ Found individual book: "3. The Secret Garden"
[6:17:48 PM] ℹ️ Processing 3 individual book sections
[6:17:48 PM] 📋 📖 PROCESSING BOOK 1: "Henry Huggins"
[6:17:48 PM] ℹ️ Expanding book section: "Henry Huggins"
[6:17:51 PM] ✅ Book section expanded: "Henry Huggins"
[6:17:51 PM] ℹ️ Expanding "Why this book is the match" section for: "Henry Huggins"
[6:17:54 PM] ✅ "Why this book is the match" section expanded for: "Henry Huggins"
[6:17:54 PM] ℹ️ Found 3 citation buttons for "Henry Huggins"
[6:17:54 PM] ℹ️ Extracting citation 1/3 for "Henry Huggins"
[6:17:54 PM] ℹ️ Processing metadata citation 1 for "Henry Huggins"
[6:17:55 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:17:57 PM] ℹ️ Extracted citation text (first 100 chars): classic...
[6:17:57 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:17:58 PM] ✅ ✅ metadata citation 1 closed successfully
[6:17:58 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 7)
[6:17:58 PM] ℹ️ Extracting citation 2/3 for "Henry Huggins"
[6:17:58 PM] ℹ️ Processing metadata citation 2 for "Henry Huggins"
[6:17:59 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:18:01 PM] ℹ️ Extracted citation text (first 100 chars): Just as Henry Huggins is complaining that nothing exciting ever happens, a friendly dog sits down be...
[6:18:01 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:18:03 PM] ✅ ✅ metadata citation 2 closed successfully
[6:18:03 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 390)
[6:18:03 PM] ℹ️ Extracting citation 3/3 for "Henry Huggins"
[6:18:03 PM] ℹ️ Processing metadata citation 3 for "Henry Huggins"
[6:18:04 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:18:06 PM] ℹ️ Extracted citation text (first 100 chars): JUVENILE FICTION: Humorous Stories...
[6:18:06 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:18:07 PM] ✅ ✅ metadata citation 3 closed successfully
[6:18:07 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 34)
[6:18:07 PM] ℹ️ Collapsing book section: "Henry Huggins"
[6:18:09 PM] ✅ Book section collapsed: "Henry Huggins"
[6:18:09 PM] ✅ ✅ Completed processing "Henry Huggins" - 3 citations extracted
[6:18:09 PM] 📋 📖 PROCESSING BOOK 2: "Anne of Green Gables"
[6:18:09 PM] ℹ️ Expanding book section: "Anne of Green Gables"
[6:18:12 PM] ✅ Book section expanded: "Anne of Green Gables"
[6:18:12 PM] ℹ️ Expanding "Why this book is the match" section for: "Anne of Green Gables"
[6:18:15 PM] ✅ "Why this book is the match" section expanded for: "Anne of Green Gables"
[6:18:15 PM] ℹ️ Found 3 citation buttons for "Anne of Green Gables"
[6:18:15 PM] ℹ️ Extracting citation 1/3 for "Anne of Green Gables"
[6:18:15 PM] ℹ️ Processing metadata citation 1 for "Anne of Green Gables"
[6:18:16 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:18:18 PM] ℹ️ Extracted citation text (first 100 chars): classic...
[6:18:18 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:18:20 PM] ✅ ✅ metadata citation 1 closed successfully
[6:18:20 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 7)
[6:18:20 PM] ℹ️ Extracting citation 2/3 for "Anne of Green Gables"
[6:18:20 PM] ℹ️ Processing metadata citation 2 for "Anne of Green Gables"
[6:18:21 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:18:23 PM] ℹ️ Extracted citation text (first 100 chars): a talkative eleven-year-old orphan with a heart full of dreams and a desperate longing for a home....
[6:18:23 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:18:24 PM] ✅ ✅ metadata citation 2 closed successfully
[6:18:24 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 98)
[6:18:24 PM] ℹ️ Extracting citation 3/3 for "Anne of Green Gables"
[6:18:24 PM] ℹ️ Processing metadata citation 3 for "Anne of Green Gables"
[6:18:25 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:18:28 PM] ℹ️ Extracted citation text (first 100 chars): JUVENILE FICTION...
[6:18:28 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:18:29 PM] ✅ ✅ metadata citation 3 closed successfully
[6:18:29 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 16)
[6:18:29 PM] ℹ️ Collapsing book section: "Anne of Green Gables"
[6:18:31 PM] ✅ Book section collapsed: "Anne of Green Gables"
[6:18:31 PM] ✅ ✅ Completed processing "Anne of Green Gables" - 3 citations extracted
[6:18:31 PM] 📋 📖 PROCESSING BOOK 3: "The Secret Garden"
[6:18:31 PM] ℹ️ Expanding book section: "The Secret Garden"
[6:18:34 PM] ✅ Book section expanded: "The Secret Garden"
[6:18:34 PM] ℹ️ Expanding "Why this book is the match" section for: "The Secret Garden"
[6:18:37 PM] ✅ "Why this book is the match" section expanded for: "The Secret Garden"
[6:18:37 PM] ℹ️ Found 3 citation buttons for "The Secret Garden"
[6:18:37 PM] ℹ️ Extracting citation 1/3 for "The Secret Garden"
[6:18:37 PM] ℹ️ Processing metadata citation 1 for "The Secret Garden"
[6:18:38 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:18:40 PM] ℹ️ Extracted citation text (first 100 chars): classic literature...
[6:18:40 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:18:41 PM] ✅ ✅ metadata citation 1 closed successfully
[6:18:41 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 18)
[6:18:41 PM] ℹ️ Extracting citation 2/3 for "The Secret Garden"
[6:18:41 PM] ℹ️ Processing metadata citation 2 for "The Secret Garden"
[6:18:42 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:18:44 PM] ℹ️ Extracted citation text (first 100 chars): Mary discovers a secret garden, surrounded by walls and locked with a missing key. One day, with the...
[6:18:44 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:18:46 PM] ✅ ✅ metadata citation 2 closed successfully
[6:18:46 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 228)
[6:18:46 PM] ℹ️ Extracting citation 3/3 for "The Secret Garden"
[6:18:46 PM] ℹ️ Processing metadata citation 3 for "The Secret Garden"
[6:18:47 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:18:49 PM] ℹ️ Extracted citation text (first 100 chars): JUVENILE FICTION...
[6:18:49 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:18:51 PM] ✅ ✅ metadata citation 3 closed successfully
[6:18:51 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 16)
[6:18:51 PM] ℹ️ Collapsing book section: "The Secret Garden"
[6:18:52 PM] ✅ Book section collapsed: "The Secret Garden"
[6:18:52 PM] ✅ ✅ Completed processing "The Secret Garden" - 3 citations extracted
[6:18:52 PM] ✅ 🎉 Successfully extracted citations from 3 books
[6:18:52 PM] ℹ️ Step 3: Validating reason-citation matches
[6:18:52 PM] 📋 🔍 VALIDATING REASON-CITATION MATCHES (80% THRESHOLD)
[6:18:52 PM] ℹ️ 📖 Validating 3 reasons for: "Henry Huggins"
[6:18:52 PM] ℹ️ Reason 1: Checking match...
[6:18:52 PM] ✅ ✅ "Henry Huggins" - Reason 1: 100% match - PASS
[6:18:52 PM] ℹ️ Reason 2: Checking match...
[6:18:59 PM] ❌ ❌ "Henry Huggins" - Reason 2: 0% match - FAIL
[6:18:59 PM] ❌ ⚠️ Citation text not found in reason: "Just as Henry Huggins is complaining that nothing exciting ever happens, a friendly dog sits down beside him and looks pleadingly at his ice-cream cone. From that moment on, the two are inseparable. But when Ribsy's original owner appears, trying to reclaim his dog, Henry's faced with the possibility of losing his new best friend. Has Klickitat Street seen the last of rambunctious Ribsy?"
[6:18:59 PM] ℹ️ Reason 3: Checking match...
[6:19:07 PM] ✅ ✅ "Henry Huggins" - Reason 3: 95% match - PASS
[6:19:07 PM] ℹ️ 📖 Validating 3 reasons for: "Anne of Green Gables"
[6:19:07 PM] ℹ️ Reason 1: Checking match...
[6:19:07 PM] ✅ ✅ "Anne of Green Gables" - Reason 1: 100% match - PASS
[6:19:07 PM] ℹ️ Reason 2: Checking match...
[6:19:07 PM] ✅ ✅ "Anne of Green Gables" - Reason 2: 100% match - PASS
[6:19:07 PM] ℹ️ Reason 3: Checking match...
[6:19:07 PM] ✅ ✅ "Anne of Green Gables" - Reason 3: 100% match - PASS
[6:19:07 PM] ℹ️ 📖 Validating 3 reasons for: "The Secret Garden"
[6:19:07 PM] ℹ️ Reason 1: Checking match...
[6:19:10 PM] ❌ ❌ "The Secret Garden" - Reason 1: 0% match - FAIL
[6:19:10 PM] ❌ ⚠️ Citation text not found in reason: "classic literature"
[6:19:10 PM] ℹ️ Reason 2: Checking match...
[6:19:25 PM] ❌ ❌ "The Secret Garden" - Reason 2: 0% match - FAIL
[6:19:25 PM] ❌ ⚠️ Citation text not found in reason: "Mary discovers a secret garden, surrounded by walls and locked with a missing key. One day, with the help of two unexpected companions, she discovers a way in. Is everything in the garden dead, or can Mary bring it back to life?"
[6:19:25 PM] ℹ️ Reason 3: Checking match...
[6:19:25 PM] ✅ ✅ "The Secret Garden" - Reason 3: 100% match - PASS
[6:19:25 PM] 📊 📊 CITATION VALIDATION SUMMARY: 6/9 passed (66.7%)
[6:19:25 PM] ❌ 💥 REASON-CITATION VALIDATION: OVERALL FAILED
[6:19:25 PM] ℹ️ Step 4: Generating detailed report
[6:19:25 PM] 📋 📋 GENERATING CITATION VALIDATION REPORT
[6:19:25 PM] ℹ️ Report generated: 6/9 reasons passed
[6:19:25 PM] ✅ ✅ Citation validation workflow completed successfully
[6:19:25 PM] 📊 📈 OVERALL CITATION VALIDATION: 6/9 (66.7%)
[6:19:25 PM] ❌ ❌ REASON-CITATION VALIDATION: OVERALL FAILED - Only 66.7% of reasons match citations (required: 80%)
[6:19:25 PM] 📋 📋 GENERATING DETAILED CITATION VALIDATION REPORT
[6:19:25 PM] ℹ️ Generating HTML citation validation report
[6:19:25 PM] ✅ Generated HTML report for 9 reasons
[6:19:25 PM] ✅ ✅ Citation validation HTML report generated and attached
[6:19:25 PM] ✅ ✅ Citation validation plain text report attached
[6:19:25 PM] ℹ️ Checking database connectivity...
[6:19:25 PM] ✅ ✅ Database connected successfully
[6:19:25 PM] ℹ️ Database path: /home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/test_data/database.xlsx
[6:19:25 PM] ℹ️ Total books: 10032
[6:19:26 PM] 📊 Database contains 10032 books
[6:19:26 PM] ✅ ✅ Database book count requirement met: 10032 >= 2020
[6:19:26 PM] 📋 🔍 VALIDATING EXTRACTED BOOKS INDIVIDUALLY AGAINST DATABASE
[6:19:26 PM] ℹ️ Checking book 1/3: "Henry Huggins"
[6:19:26 PM] ℹ️ Checking if book exists in database: "Henry Huggins"
[6:19:26 PM] ✅ ✅ Database match: "Henry Huggins" → "Henry Huggins 4-Book Collection"
[6:19:26 PM] ℹ️ Checking book 2/3: "Anne of Green Gables"
[6:19:26 PM] ℹ️ Checking if book exists in database: "Anne of Green Gables"
[6:19:26 PM] ✅ ✅ Database match: "Anne of Green Gables" → "Anne of Green Gables"
[6:19:26 PM] ℹ️ Checking book 3/3: "The Secret Garden"
[6:19:26 PM] ℹ️ Checking if book exists in database: "The Secret Garden"
[6:19:26 PM] ✅ ✅ Database match: "The Secret Garden" → "The Secret Garden"
[6:19:26 PM] 📊 INDIVIDUAL DATABASE VALIDATION SUMMARY: 3/3 books found in database
[6:19:26 PM] ✅ 🎉 ALL BOOKS VALIDATED SUCCESSFULLY AGAINST DATABASE!
[6:19:26 PM] 📋 🤖 VALIDATING RESPONSE RELEVANCE WITH GEMINI AI - PER BOOK ANALYSIS
[6:19:26 PM] ℹ️ Starting Gemini AI per-book relevance validation...
[6:19:26 PM] 📋 🤖 ANALYZING RESPONSE RELEVANCE WITH GEMINI AI - PER BOOK ANALYSIS
[6:19:26 PM] ℹ️ Starting Gemini AI analysis with retry logic...
[6:19:32 PM] ⚠ ⚠️ GEMINI VALIDATION: Using fallback analysis due to API issues
[6:19:32 PM] ℹ️ Overall Score: 70% (Fallback)
[6:19:32 PM] 📋 📖 ANALYSIS FOR: "Henry Huggins" - Score: 70%
[6:19:32 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ 💡 Book-specific improvements:
[6:19:32 PM] ℹ️ - Retry analysis when service is available
[6:19:32 PM] 📋 📖 ANALYSIS FOR: "Anne of Green Gables" - Score: 70%
[6:19:32 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ 💡 Book-specific improvements:
[6:19:32 PM] ℹ️ - Retry analysis when service is available
[6:19:32 PM] 📋 📖 ANALYSIS FOR: "The Secret Garden" - Score: 70%
[6:19:32 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ 💡 Book-specific improvements:
[6:19:32 PM] ℹ️ - Retry analysis when service is available
[6:19:32 PM] ℹ️ 📊 SUMMARY FEEDBACK:
[6:19:32 PM] ℹ️ • AI analysis service is temporarily overloaded
[6:19:32 PM] ℹ️ • Using fallback analysis with neutral scores
[6:19:32 PM] ℹ️ • Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits.
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2
Please retry in 27.101040378s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"27s"}]
[6:19:32 PM] ❌ ❌ GEMINI AI VALIDATION FAILED: 70% relevance score (required: 80%)
[6:19:32 PM] ℹ️ Continuing execution due to soft assertion...
=== END LOGS ===
Scenario Logs: Book Genie Response Validation with Citation Verification
[6:17:27 PM] Navigating to Creative Workspace login page
[6:17:30 PM] Homepage visibility: false
[6:17:30 PM] ℹ️ Typing query: "Suggest 3 books for childrens"
[6:17:30 PM] ℹ️ Preparing to type query: "Suggest 3 books for childrens"
[6:17:30 PM] ℹ️ Waiting for chat input to be visible
[6:17:32 PM] ℹ️ Clearing chat input
[6:17:32 PM] ℹ️ Typing query: "Suggest 3 books for childrens"
[6:17:33 PM] ℹ️ Pressing Enter to submit query
[6:17:33 PM] ✅ Query submitted successfully: "Suggest 3 books for childrens"
[6:17:33 PM] ℹ️ Waiting for AI to complete thinking...
[6:17:33 PM] 📋 WAITING FOR AI RESPONSE
[6:17:33 PM] ℹ️ Looking for AI thinking indicator...
[6:17:33 PM] ℹ️ Waiting for thinking indicator to appear (max 2 minutes)
[6:17:34 PM] ✅ ✓ AI thinking indicator appeared - AI is processing the request
[6:17:34 PM] ℹ️ Waiting for thinking indicator to disappear (max 5 minutes)
[6:17:44 PM] ✅ ✓ AI thinking completed - Response should be ready
[6:17:44 PM] ℹ️ Allowing additional time for response rendering
[6:17:47 PM] ✅ AI response wait process completed
[6:17:47 PM] ℹ️ Validating response for query: "Suggest 3 books for childrens"
[6:17:47 PM] 📋 VALIDATING RESPONSE FOR QUERY: "Suggest 3 books for childrens"
[6:17:47 PM] ℹ️ Locating the latest response in chat
[6:17:47 PM] ℹ️ Waiting for response to be visible (max 30 seconds)
[6:17:47 PM] ✅ Response received - Length: 2030 characters
[6:17:47 PM] ℹ️ Response preview: Watch Me WorkInterpreting context for your query...Retrieving Relevant Books for 'books suitable for...
[6:17:47 PM] ℹ️ Extracting book data from response...
[6:17:47 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:17:47 PM] ℹ️ Locating BookGenie response container
[6:17:47 PM] ℹ️ Extracting HTML content from response
[6:17:47 PM] ℹ️ HTML content extracted - Length: 25302 characters
[6:17:47 PM] ℹ️ Parsing HTML to extract book data
[6:17:47 PM] ✅ SUCCESSFULLY EXTRACTED 3 BOOKS
[6:17:47 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:17:47 PM] 📚 1. "Henry Huggins" - Score: 100%%
[6:17:47 PM] 📚 2. "Anne of Green Gables" - Score: 100%%
[6:17:47 PM] 📚 3. "The Secret Garden" - Score: 100%%
[6:17:47 PM] 📊 Total books extracted: 3
[6:17:47 PM] 📊 Average relevance score: 100%
[6:17:47 PM] ✅ Extracted 3 books from response
[6:17:47 PM] ℹ️ Saving 3 books to Excel file: test_results/suggest_3_books_for_childrens.xlsx
[6:17:47 PM] 📋 SAVING BOOKS TO EXCEL FILE
[6:17:47 PM] ℹ️ Target file path: test_results/suggest_3_books_for_childrens.xlsx
[6:17:47 PM] ℹ️ Exporting 3 books to Excel format
[6:17:47 PM] ✅ Book data saved to: test_results/suggest_3_books_for_childrens.xlsx
[6:17:47 PM] 📊 File verified - Size: 16.87 KB
[6:17:47 PM] ✅ ✓ Excel file creation verified successfully
[6:17:47 PM] 📋 STARTING INDIVIDUAL BOOK VALIDATION (3 books) for query: "Suggest 3 books for childrens"
[6:17:47 PM] 📋 📚 VALIDATING BOOK 1: "Henry Huggins"
[6:17:47 PM] ℹ️ Starting comprehensive book validation
[6:17:47 PM] ℹ️ Validating title: "Henry Huggins" vs expected: "Henry Huggins"
[6:17:47 PM] ✅ ✅ TITLE VALIDATION PASSED: "Henry Huggins"
[6:17:47 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:17:47 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:17:47 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:17:47 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:17:47 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:17:47 PM] ✅ 🎉 BOOK 1 VALIDATION: ALL CHECKS PASSED!
[6:17:47 PM] ℹ️ Book 1 validation completed
[6:17:47 PM] 📋 📚 VALIDATING BOOK 2: "Anne of Green Gables"
[6:17:47 PM] ℹ️ Starting comprehensive book validation
[6:17:47 PM] ℹ️ Validating title: "Anne of Green Gables" vs expected: "Anne of Green Gables"
[6:17:47 PM] ✅ ✅ TITLE VALIDATION PASSED: "Anne of Green Gables"
[6:17:47 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:17:47 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:17:47 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:17:47 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:17:47 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:17:47 PM] ✅ 🎉 BOOK 2 VALIDATION: ALL CHECKS PASSED!
[6:17:47 PM] ℹ️ Book 2 validation completed
[6:17:47 PM] 📋 📚 VALIDATING BOOK 3: "The Secret Garden"
[6:17:47 PM] ℹ️ Starting comprehensive book validation
[6:17:47 PM] ℹ️ Validating title: "The Secret Garden" vs expected: "The Secret Garden"
[6:17:47 PM] ✅ ✅ TITLE VALIDATION PASSED: "The Secret Garden"
[6:17:47 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:17:47 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:17:47 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:17:47 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:17:47 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:17:47 PM] ✅ 🎉 BOOK 3 VALIDATION: ALL CHECKS PASSED!
[6:17:47 PM] ℹ️ Book 3 validation completed
[6:17:47 PM] 📊 VALIDATION SUMMARY: 3/3 books passed
[6:17:47 PM] ✅ ALL BOOKS VALIDATED SUCCESSFULLY!
[6:17:47 PM] 📋 🔍 VALIDATING REASON-CITATION MATCHES
[6:17:47 PM] ℹ️ Starting citation validation process...
[6:17:47 PM] 📋 🚀 STARTING COMPLETE CITATION VALIDATION WORKFLOW
[6:17:47 PM] ℹ️ Step 1: Extracting books from AI response
[6:17:47 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:17:47 PM] ℹ️ Locating BookGenie response container
[6:17:47 PM] ℹ️ Extracting HTML content from response
[6:17:47 PM] ℹ️ HTML content extracted - Length: 25302 characters
[6:17:47 PM] ℹ️ Parsing HTML to extract book data
[6:17:47 PM] ✅ SUCCESSFULLY EXTRACTED 3 BOOKS
[6:17:47 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:17:47 PM] 📚 1. "Henry Huggins" - Score: 100%%
[6:17:47 PM] 📚 2. "Anne of Green Gables" - Score: 100%%
[6:17:47 PM] 📚 3. "The Secret Garden" - Score: 100%%
[6:17:47 PM] 📊 Total books extracted: 3
[6:17:47 PM] 📊 Average relevance score: 100%
[6:17:47 PM] ℹ️ Step 2: Extracting citation texts from book elements
[6:17:47 PM] 📋 📚 EXTRACTING CITATION TEXTS FROM BOOK ELEMENTS
[6:17:47 PM] ℹ️ Step 1: Waiting for main book section
[6:17:47 PM] ℹ️ Found 7 total accordion sections
[6:17:47 PM] ℹ️ Found individual book: "1. Henry Huggins"
[6:17:47 PM] ℹ️ Found individual book: "2. Anne of Green Gables"
[6:17:48 PM] ℹ️ Found individual book: "3. The Secret Garden"
[6:17:48 PM] ℹ️ Processing 3 individual book sections
[6:17:48 PM] 📋 📖 PROCESSING BOOK 1: "Henry Huggins"
[6:17:48 PM] ℹ️ Expanding book section: "Henry Huggins"
[6:17:51 PM] ✅ Book section expanded: "Henry Huggins"
[6:17:51 PM] ℹ️ Expanding "Why this book is the match" section for: "Henry Huggins"
[6:17:54 PM] ✅ "Why this book is the match" section expanded for: "Henry Huggins"
[6:17:54 PM] ℹ️ Found 3 citation buttons for "Henry Huggins"
[6:17:54 PM] ℹ️ Extracting citation 1/3 for "Henry Huggins"
[6:17:54 PM] ℹ️ Processing metadata citation 1 for "Henry Huggins"
[6:17:55 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:17:57 PM] ℹ️ Extracted citation text (first 100 chars): classic...
[6:17:57 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:17:58 PM] ✅ ✅ metadata citation 1 closed successfully
[6:17:58 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 7)
[6:17:58 PM] ℹ️ Extracting citation 2/3 for "Henry Huggins"
[6:17:58 PM] ℹ️ Processing metadata citation 2 for "Henry Huggins"
[6:17:59 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:18:01 PM] ℹ️ Extracted citation text (first 100 chars): Just as Henry Huggins is complaining that nothing exciting ever happens, a friendly dog sits down be...
[6:18:01 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:18:03 PM] ✅ ✅ metadata citation 2 closed successfully
[6:18:03 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 390)
[6:18:03 PM] ℹ️ Extracting citation 3/3 for "Henry Huggins"
[6:18:03 PM] ℹ️ Processing metadata citation 3 for "Henry Huggins"
[6:18:04 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:18:06 PM] ℹ️ Extracted citation text (first 100 chars): JUVENILE FICTION: Humorous Stories...
[6:18:06 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:18:07 PM] ✅ ✅ metadata citation 3 closed successfully
[6:18:07 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 34)
[6:18:07 PM] ℹ️ Collapsing book section: "Henry Huggins"
[6:18:09 PM] ✅ Book section collapsed: "Henry Huggins"
[6:18:09 PM] ✅ ✅ Completed processing "Henry Huggins" - 3 citations extracted
[6:18:09 PM] 📋 📖 PROCESSING BOOK 2: "Anne of Green Gables"
[6:18:09 PM] ℹ️ Expanding book section: "Anne of Green Gables"
[6:18:12 PM] ✅ Book section expanded: "Anne of Green Gables"
[6:18:12 PM] ℹ️ Expanding "Why this book is the match" section for: "Anne of Green Gables"
[6:18:15 PM] ✅ "Why this book is the match" section expanded for: "Anne of Green Gables"
[6:18:15 PM] ℹ️ Found 3 citation buttons for "Anne of Green Gables"
[6:18:15 PM] ℹ️ Extracting citation 1/3 for "Anne of Green Gables"
[6:18:15 PM] ℹ️ Processing metadata citation 1 for "Anne of Green Gables"
[6:18:16 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:18:18 PM] ℹ️ Extracted citation text (first 100 chars): classic...
[6:18:18 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:18:20 PM] ✅ ✅ metadata citation 1 closed successfully
[6:18:20 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 7)
[6:18:20 PM] ℹ️ Extracting citation 2/3 for "Anne of Green Gables"
[6:18:20 PM] ℹ️ Processing metadata citation 2 for "Anne of Green Gables"
[6:18:21 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:18:23 PM] ℹ️ Extracted citation text (first 100 chars): a talkative eleven-year-old orphan with a heart full of dreams and a desperate longing for a home....
[6:18:23 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:18:24 PM] ✅ ✅ metadata citation 2 closed successfully
[6:18:24 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 98)
[6:18:24 PM] ℹ️ Extracting citation 3/3 for "Anne of Green Gables"
[6:18:24 PM] ℹ️ Processing metadata citation 3 for "Anne of Green Gables"
[6:18:25 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:18:28 PM] ℹ️ Extracted citation text (first 100 chars): JUVENILE FICTION...
[6:18:28 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:18:29 PM] ✅ ✅ metadata citation 3 closed successfully
[6:18:29 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 16)
[6:18:29 PM] ℹ️ Collapsing book section: "Anne of Green Gables"
[6:18:31 PM] ✅ Book section collapsed: "Anne of Green Gables"
[6:18:31 PM] ✅ ✅ Completed processing "Anne of Green Gables" - 3 citations extracted
[6:18:31 PM] 📋 📖 PROCESSING BOOK 3: "The Secret Garden"
[6:18:31 PM] ℹ️ Expanding book section: "The Secret Garden"
[6:18:34 PM] ✅ Book section expanded: "The Secret Garden"
[6:18:34 PM] ℹ️ Expanding "Why this book is the match" section for: "The Secret Garden"
[6:18:37 PM] ✅ "Why this book is the match" section expanded for: "The Secret Garden"
[6:18:37 PM] ℹ️ Found 3 citation buttons for "The Secret Garden"
[6:18:37 PM] ℹ️ Extracting citation 1/3 for "The Secret Garden"
[6:18:37 PM] ℹ️ Processing metadata citation 1 for "The Secret Garden"
[6:18:38 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:18:40 PM] ℹ️ Extracted citation text (first 100 chars): classic literature...
[6:18:40 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:18:41 PM] ✅ ✅ metadata citation 1 closed successfully
[6:18:41 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 18)
[6:18:41 PM] ℹ️ Extracting citation 2/3 for "The Secret Garden"
[6:18:41 PM] ℹ️ Processing metadata citation 2 for "The Secret Garden"
[6:18:42 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:18:44 PM] ℹ️ Extracted citation text (first 100 chars): Mary discovers a secret garden, surrounded by walls and locked with a missing key. One day, with the...
[6:18:44 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:18:46 PM] ✅ ✅ metadata citation 2 closed successfully
[6:18:46 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 228)
[6:18:46 PM] ℹ️ Extracting citation 3/3 for "The Secret Garden"
[6:18:46 PM] ℹ️ Processing metadata citation 3 for "The Secret Garden"
[6:18:47 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:18:49 PM] ℹ️ Extracted citation text (first 100 chars): JUVENILE FICTION...
[6:18:49 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:18:51 PM] ✅ ✅ metadata citation 3 closed successfully
[6:18:51 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 16)
[6:18:51 PM] ℹ️ Collapsing book section: "The Secret Garden"
[6:18:52 PM] ✅ Book section collapsed: "The Secret Garden"
[6:18:52 PM] ✅ ✅ Completed processing "The Secret Garden" - 3 citations extracted
[6:18:52 PM] ✅ 🎉 Successfully extracted citations from 3 books
[6:18:52 PM] ℹ️ Step 3: Validating reason-citation matches
[6:18:52 PM] 📋 🔍 VALIDATING REASON-CITATION MATCHES (80% THRESHOLD)
[6:18:52 PM] ℹ️ 📖 Validating 3 reasons for: "Henry Huggins"
[6:18:52 PM] ℹ️ Reason 1: Checking match...
[6:18:52 PM] ✅ ✅ "Henry Huggins" - Reason 1: 100% match - PASS
[6:18:52 PM] ℹ️ Reason 2: Checking match...
[6:18:59 PM] ❌ ❌ "Henry Huggins" - Reason 2: 0% match - FAIL
[6:18:59 PM] ❌ ⚠️ Citation text not found in reason: "Just as Henry Huggins is complaining that nothing exciting ever happens, a friendly dog sits down beside him and looks pleadingly at his ice-cream cone. From that moment on, the two are inseparable. But when Ribsy's original owner appears, trying to reclaim his dog, Henry's faced with the possibility of losing his new best friend. Has Klickitat Street seen the last of rambunctious Ribsy?"
[6:18:59 PM] ℹ️ Reason 3: Checking match...
[6:19:07 PM] ✅ ✅ "Henry Huggins" - Reason 3: 95% match - PASS
[6:19:07 PM] ℹ️ 📖 Validating 3 reasons for: "Anne of Green Gables"
[6:19:07 PM] ℹ️ Reason 1: Checking match...
[6:19:07 PM] ✅ ✅ "Anne of Green Gables" - Reason 1: 100% match - PASS
[6:19:07 PM] ℹ️ Reason 2: Checking match...
[6:19:07 PM] ✅ ✅ "Anne of Green Gables" - Reason 2: 100% match - PASS
[6:19:07 PM] ℹ️ Reason 3: Checking match...
[6:19:07 PM] ✅ ✅ "Anne of Green Gables" - Reason 3: 100% match - PASS
[6:19:07 PM] ℹ️ 📖 Validating 3 reasons for: "The Secret Garden"
[6:19:07 PM] ℹ️ Reason 1: Checking match...
[6:19:10 PM] ❌ ❌ "The Secret Garden" - Reason 1: 0% match - FAIL
[6:19:10 PM] ❌ ⚠️ Citation text not found in reason: "classic literature"
[6:19:10 PM] ℹ️ Reason 2: Checking match...
[6:19:25 PM] ❌ ❌ "The Secret Garden" - Reason 2: 0% match - FAIL
[6:19:25 PM] ❌ ⚠️ Citation text not found in reason: "Mary discovers a secret garden, surrounded by walls and locked with a missing key. One day, with the help of two unexpected companions, she discovers a way in. Is everything in the garden dead, or can Mary bring it back to life?"
[6:19:25 PM] ℹ️ Reason 3: Checking match...
[6:19:25 PM] ✅ ✅ "The Secret Garden" - Reason 3: 100% match - PASS
[6:19:25 PM] 📊 📊 CITATION VALIDATION SUMMARY: 6/9 passed (66.7%)
[6:19:25 PM] ❌ 💥 REASON-CITATION VALIDATION: OVERALL FAILED
[6:19:25 PM] ℹ️ Step 4: Generating detailed report
[6:19:25 PM] 📋 📋 GENERATING CITATION VALIDATION REPORT
[6:19:25 PM] ℹ️ Report generated: 6/9 reasons passed
[6:19:25 PM] ✅ ✅ Citation validation workflow completed successfully
[6:19:25 PM] 📊 📈 OVERALL CITATION VALIDATION: 6/9 (66.7%)
[6:19:25 PM] ❌ ❌ REASON-CITATION VALIDATION: OVERALL FAILED - Only 66.7% of reasons match citations (required: 80%)
[6:19:25 PM] 📋 📋 GENERATING DETAILED CITATION VALIDATION REPORT
[6:19:25 PM] ℹ️ Generating HTML citation validation report
[6:19:25 PM] ✅ Generated HTML report for 9 reasons
[6:19:25 PM] ✅ ✅ Citation validation HTML report generated and attached
[6:19:25 PM] ✅ ✅ Citation validation plain text report attached
[6:19:25 PM] ℹ️ Checking database connectivity...
[6:19:25 PM] ✅ ✅ Database connected successfully
[6:19:25 PM] ℹ️ Database path: /home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/test_data/database.xlsx
[6:19:25 PM] ℹ️ Total books: 10032
[6:19:26 PM] 📊 Database contains 10032 books
[6:19:26 PM] ✅ ✅ Database book count requirement met: 10032 >= 2020
[6:19:26 PM] 📋 🔍 VALIDATING EXTRACTED BOOKS INDIVIDUALLY AGAINST DATABASE
[6:19:26 PM] ℹ️ Checking book 1/3: "Henry Huggins"
[6:19:26 PM] ℹ️ Checking if book exists in database: "Henry Huggins"
[6:19:26 PM] ✅ ✅ Database match: "Henry Huggins" → "Henry Huggins 4-Book Collection"
[6:19:26 PM] ℹ️ Checking book 2/3: "Anne of Green Gables"
[6:19:26 PM] ℹ️ Checking if book exists in database: "Anne of Green Gables"
[6:19:26 PM] ✅ ✅ Database match: "Anne of Green Gables" → "Anne of Green Gables"
[6:19:26 PM] ℹ️ Checking book 3/3: "The Secret Garden"
[6:19:26 PM] ℹ️ Checking if book exists in database: "The Secret Garden"
[6:19:26 PM] ✅ ✅ Database match: "The Secret Garden" → "The Secret Garden"
[6:19:26 PM] 📊 INDIVIDUAL DATABASE VALIDATION SUMMARY: 3/3 books found in database
[6:19:26 PM] ✅ 🎉 ALL BOOKS VALIDATED SUCCESSFULLY AGAINST DATABASE!
[6:19:26 PM] 📋 🤖 VALIDATING RESPONSE RELEVANCE WITH GEMINI AI - PER BOOK ANALYSIS
[6:19:26 PM] ℹ️ Starting Gemini AI per-book relevance validation...
[6:19:26 PM] 📋 🤖 ANALYZING RESPONSE RELEVANCE WITH GEMINI AI - PER BOOK ANALYSIS
[6:19:26 PM] ℹ️ Starting Gemini AI analysis with retry logic...
[6:19:32 PM] ⚠ ⚠️ GEMINI VALIDATION: Using fallback analysis due to API issues
[6:19:32 PM] ℹ️ Overall Score: 70% (Fallback)
[6:19:32 PM] 📋 📖 ANALYSIS FOR: "Henry Huggins" - Score: 70%
[6:19:32 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ 💡 Book-specific improvements:
[6:19:32 PM] ℹ️ - Retry analysis when service is available
[6:19:32 PM] 📋 📖 ANALYSIS FOR: "Anne of Green Gables" - Score: 70%
[6:19:32 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ 💡 Book-specific improvements:
[6:19:32 PM] ℹ️ - Retry analysis when service is available
[6:19:32 PM] 📋 📖 ANALYSIS FOR: "The Secret Garden" - Score: 70%
[6:19:32 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:19:32 PM] ℹ️ 💡 Book-specific improvements:
[6:19:32 PM] ℹ️ - Retry analysis when service is available
[6:19:32 PM] ℹ️ 📊 SUMMARY FEEDBACK:
[6:19:32 PM] ℹ️ • AI analysis service is temporarily overloaded
[6:19:32 PM] ℹ️ • Using fallback analysis with neutral scores
[6:19:32 PM] ℹ️ • Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits.
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2
Please retry in 27.101040378s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"27s"}]
[6:19:32 PM] ❌ ❌ GEMINI AI VALIDATION FAILED: 70% relevance score (required: 80%)
[6:19:32 PM] ℹ️ Continuing execution due to soft assertion...